{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e1c2b0e7",
   "metadata": {},
   "source": [
    "可以在[Bookshop.org](https://bookshop.org/a/98697/9781098155438) 和\n",
    "[Amazon](https://www.amazon.com/_/dp/1098155432?smid=ATVPDKIKX0DER&_encoding=UTF8&tag=oreilly20-20&_encoding=UTF8&tag=greenteapre01-20&linkCode=ur2&linkId=e2a529f94920295d27ec8a06e757dc7c&camp=1789&creative=9325)获取纸制版和电子版的*Think Python 3e*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "31b6f97a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print(\"Downloaded \" + str(local))\n",
    "    return filename\n",
    "\n",
    "download('https://gitee.com/regentsai/Think_Python_3e_CN/blob/master/thinkpython.py');\n",
    "download('https://gitee.com/regentsai/Think_Python_3e_CN/blob/master/diagram.py');\n",
    "\n",
    "import thinkpython"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3ee5c27",
   "metadata": {
    "tags": []
   },
   "source": [
    "致谢：本章图片来自[Lorem Picsum](https://picsum.photos/)的占位图片。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9b246dc6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 下载本章使用的示例文件\n",
    "\n",
    "download('https://gitee.com/regentsai/Think_Python_3e_CN/blob/master/photos.zip');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "23792899",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 删除已经存在的photos/文件夹\n",
    "# 重复运行本章笔记需要运行\n",
    "\n",
    "# !rm -rf photos/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "713dfeb0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  photos.zip\n",
      "  inflating: photos/notes.txt        \n",
      "  inflating: photos/mar-2023/photo2.jpg  \n",
      "  inflating: photos/mar-2023/photo1.jpg  \n",
      "  inflating: photos/jan-2023/photo3.jpg  \n",
      "  inflating: photos/jan-2023/photo2.jpg  \n",
      "  inflating: photos/jan-2023/photo1.jpg  \n",
      "  inflating: photos/feb-2023/photo2.jpg  \n",
      "  inflating: photos/feb-2023/photo1.jpg  \n"
     ]
    }
   ],
   "source": [
    "!unzip -o photos.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92934cb0",
   "metadata": {},
   "source": [
    "# 第13章:文件和数据库\n",
    "\n",
    "我们目前看到的多数程序是**临时 ephemeral**的，因为它们运行一会儿，然后产生输出，当程序结束后，这些数据会消失。每当你运行临时的程序时，它将从头开始。\n",
    "\n",
    "其他程序是**持久 persistent**的，它们运行很长时间（或者一直运行）;他们长期存储至少一部分数据；如果它们停止并重启，它们从离开的状态继续运行。\n",
    "\n",
    "要保持程序的数据，读取和写入文本文件是一种简单的方法。更复杂的方式是存储到数据库中。数据库是特殊的文件，可以更有效地读写数据，并提供额外的功能。\n",
    "\n",
    "本章我们将编写程序，读写文本文件和数据库，也将编写程序搜索图库中的重复文件。但在你处理文件之前，你必须找到它，所以我们将从文件名，路径，目录开始。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75cec7ca",
   "metadata": {},
   "source": [
    "## 文件名和路径\n",
    "\n",
    "文件由**目录 directories**(,也称**文件夹folders**)组织。每次运行程序时会有一个**当前工作目录 current working directory**,是多数操作发生的默认目录。例如当你要打开文件时,python将在当前工作目录查找该文件。\n",
    "\n",
    "`os`模块提供处理文件和目录的函数(os表示操作系统operating system)。其中`getcwd`函数返回当前工作目录的名字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4fdb528a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 此单元格替换`os.cwd`为返回假路径的函数\n",
    "# 译注:使用与python内建对象同名的对象是一个危险的行为,本章仅出于演示目的\n",
    "import os\n",
    "\n",
    "def getcwd():\n",
    "    return \"/home/dinsdale\"\n",
    "\n",
    "os.getcwd = getcwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c7b209f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/dinsdale'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c55575f",
   "metadata": {},
   "source": [
    "这个例子的结果显示用户名为`dinsdale`的家目录。像`'/home/dinsdale'`这样标识文件或目录的字符串称为**路径path**。\n",
    "\n",
    "一个简单的文件名，例如`'memo.txt'`也被认为是一个路径，但它是**相对路径relative path**。在这个例子中，当前目录为`/home/dinsdale`，所以`'memo.txt'`和完整的路径`'/home/dinsdale/memo.txt'`是等效的。\n",
    "\n",
    "以`/`开始的路径不依赖于当前目录，被称为**绝对路径 absolute path**。你可以使用os模块中的`abspath`函数获取一个文件的绝对路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "66a15e44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/dinsdale/memo.txt'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.abspath('memo.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1a0cf04",
   "metadata": {},
   "source": [
    "`os`模块提供其他处理文件名和路径的函数。`listdir`返回指定目录下的内容列表，包括文件和其他目录。以下是一个例子，列出了`photos`目录下的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a22d2675",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['notes.txt', 'feb-2023', 'mar-2023', 'jan-2023']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir('photos')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab160f29",
   "metadata": {},
   "source": [
    "这个目录包含一个文本文件`notes.txt`以及3个目录，目录中包含jpg格式的图像文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a217eac7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['photo3.jpg', 'photo1.jpg', 'photo2.jpg']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir('photos/jan-2023')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06b27c85",
   "metadata": {},
   "source": [
    "要检查是否存在某个文件或目录，可以使用`os.path.exists`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "aa56aeac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.exists('photos')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9049009a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.exists('photos/apr-2023')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d41f235c",
   "metadata": {},
   "source": [
    "要检查一个路径是否指向一个文件或者目录，我们可以使用`isdir`函数，当它指向一个目录则返回`True`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0feddb06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.isdir('photos')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd70cc9",
   "metadata": {},
   "source": [
    "`isfile`则在路径指向文件时返回`True`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e9f6a762",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.isfile('photos/notes.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6933c18",
   "metadata": {},
   "source": [
    "处理路径的挑战之一在于不同的操作系统上它们看起来不一样。\n",
    "\n",
    "在macOS和UNIX类系统上，路径中的目录和文件名用正斜杠`/`分隔，Windows则使用反斜杠`\\`。所以如果你在Windows上运行这些例子，你会看到路径中出现反斜杠，你必须替换示例中的正斜杠。\n",
    "\n",
    "如果要编写适合多个系统的代码，你可以使用`os.path.join`函数，将目录和文件拼接为当前操作系统上合法的路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "eb738376",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'photos/jan-2023/photo1.jpg'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.join('photos', 'jan-2023', 'photo1.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a665cf9",
   "metadata": {},
   "source": [
    "在后文中我们将使用这些函数搜索一些目录，寻找所有图像文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a96ba4",
   "metadata": {},
   "source": [
    "## f-字符串\n",
    "\n",
    "程序存储数据的方式之一是将其写成文本文件。例如，假设你是一个骆驼观察员，你想要记录在一段观察周期内看到的骆驼数。假设在1.5年内，你观察了23只骆驼，相关数据如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "23e7a6a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_years = 1.5\n",
    "num_camels = 23"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f652aaac",
   "metadata": {},
   "source": [
    "要将这些数据写入文件，你可以使用文件对象的`write`方法（见第8章）。`write`方法的参数必须是字符串，所以如果你想要将数据写入文件，就必须转化为字符串。最简单的转化方式是使用内建的函数`str`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "174d4f83",
   "metadata": {},
   "outputs": [],
   "source": [
    "writer = open('camel-spotting-book.txt', 'w')\n",
    "writer.write(str(num_years))\n",
    "writer.write(str(num_camels))\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "307c22d2",
   "metadata": {},
   "source": [
    "这种方式是有效的，但`write`方法不会添加空格或者换行符，除非你显式包含这些字符。当读取文件时，我们会发现这两个数字靠在一起。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5209eda3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.523'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('camel-spotting-book.txt').read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8008ecdc",
   "metadata": {},
   "source": [
    "至少我们需要在数字间添加空格。同时，让我们添加一些解释性的文本。\n",
    "\n",
    "要进行字符串与其他值的结合，我们可以使用**f字符串 f-string**，这种字符串的引号之前用`f`表示，并包含若干花括号包围的Python表达式。\n",
    "\n",
    "以下f字符串包含一个表达式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0f202d66",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'我观察了23只骆驼。'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f'我观察了{num_camels}只骆驼。'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3fb0556",
   "metadata": {},
   "source": [
    "结果是一个字符串，其中的表达式被求值，然后替换结果。可以有多个表达式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "33c5f77f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'在1.5年内，我观察了23只骆驼。'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f'在{num_years}年内，我观察了{num_camels}只骆驼。'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bace1539",
   "metadata": {},
   "source": [
    "表达式也可以包含运算符和函数调用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1fe7111c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'在18月内，我观察了23只骆驼。'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = f'在{round(num_years * 12)}月内，我观察了{num_camels}只骆驼。'\n",
    "line"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc6fa094",
   "metadata": {},
   "source": [
    "所以我们可以像这样将数据写入文本文件："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bc06e90a",
   "metadata": {},
   "outputs": [],
   "source": [
    "writer = open('camel-spotting-book.txt', 'w')\n",
    "writer.write(f'Years of observation: {num_years}\\n')\n",
    "writer.write(f'Camels spotted: {num_camels}\\n')\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f916d561",
   "metadata": {},
   "source": [
    "以`\\n`结尾的f字符串将添加一个换行符。我们可以像这样读取文件："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8d9eaf8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Years of observation: 1.5\n",
      "Camels spotted: 23\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data = open('camel-spotting-book.txt').read()\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3bf6913",
   "metadata": {},
   "source": [
    "对于f字符串，花括号中的表达式被转化为字符串，所以你可以包含列表，字典和其他类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ae3060c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Here is a list [1, 2, 3] and a dictionary {'one': 1}\""
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = [1, 2, 3]\n",
    "d = {'one': 1}\n",
    "f'Here is a list {t} and a dictionary {d}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "838ef132",
   "metadata": {
    "tags": []
   },
   "source": [
    "如果一个f字符串包含无效的表达式，将产生错误。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "26ca3d1b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "can only concatenate list (not \"int\") to list",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31mTypeError\u001b[0m\u001b[0;31m:\u001b[0m can only concatenate list (not \"int\") to list\n"
     ]
    }
   ],
   "source": [
    "%%expect TypeError\n",
    "\n",
    "f'This is not a valid expression {t + 2}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbbaaed3",
   "metadata": {},
   "source": [
    "## YAML\n",
    "\n",
    "程序读写文件的目的之一是存储**配置文件 configuration data**，控制程序如何运行。\n",
    "\n",
    "例如，在搜索重复图片的程序中，我们也许有一个字典`config`，包含要搜索的目录，要存储结果的另一个目录，以及要识别为图像文件的拓展名列表。\n",
    "\n",
    "它可能长这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5bb69e1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    'photo_dir': 'photos',\n",
    "    'data_dir': 'photo_info',\n",
    "    'extensions': ['jpg', 'jpeg'],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1667bb96",
   "metadata": {},
   "source": [
    "我们可以使用f字符串将其写入文本文件，但使用`yaml`模块会更简单，它是专为这种结构而设计的。\n",
    "\n",
    "`yaml`模块提供处理YAML文件的函数，YAML文件是方便人类和程序读写的文本文件格式。\n",
    "\n",
    "以下是使用`dump`函数将`config`字典写入YAML文件的示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0090a29f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 如果没有yaml,下载pyyaml模块\n",
    "\n",
    "try:\n",
    "    import yaml\n",
    "except ImportError:\n",
    "    !pip install pyyaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "97079479",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "config_filename = 'config.yaml'\n",
    "writer = open(config_filename, 'w')\n",
    "yaml.dump(config, writer)\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92d1b7ff",
   "metadata": {},
   "source": [
    "如果我们读取文件中的内容，YAML格式的文本如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "8646dd2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data_dir: photo_info\n",
      "extensions:\n",
      "- jpg\n",
      "- jpeg\n",
      "photo_dir: photos\n",
      "\n"
     ]
    }
   ],
   "source": [
    "readback = open(config_filename).read()\n",
    "print(readback)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33cdfd2c",
   "metadata": {},
   "source": [
    "现在我们可以使用`safe_load`从YAML文件中恢复数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e8ce2e4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data_dir': 'photo_info',\n",
       " 'extensions': ['jpg', 'jpeg'],\n",
       " 'photo_dir': 'photos'}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reader = open(config_filename)\n",
    "config_readback = yaml.safe_load(reader)\n",
    "config_readback"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca55764f",
   "metadata": {},
   "source": [
    "结果是一个新的字典，包含与原始字典相同的信息，但它们不是相同的字典。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "871d6ad5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config is config_readback"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "969ad306",
   "metadata": {},
   "source": [
    "将类似字典的对象转化为字符串的操作称为**序列化serialization**，相反的过程称为**反序列化deserialization**。如果你对一个对象进行了序列化和反序列化，得到的结果应该与原始对象相等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e130cf8",
   "metadata": {},
   "source": [
    "## Shelve\n",
    "\n",
    "目前我们已经将数据读写至文本文件，现在让我们考虑数据库。**数据库database**是一个文件，专用于存储数据。有些数据库像一张表格，在行和列上存储信息。其他数据库像一个字典，映射键到对应的值，被称为**键值存储 key-value stores**。\n",
    "\n",
    "`shelve`模块提供函数，创建和更新一个键值存储，该存储被称为\"货架shelf\"。例如，我们将创建一个货架，包含`photos`目录中图片的标题。我们将使用字典`config`获得目录的名字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "70a6d625",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'photo_info'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config['data_dir']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e6cfb65",
   "metadata": {},
   "source": [
    "我们可以使用`os.makedirs`，如果该目录不存在则创建目录。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "fd070fa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(config['data_dir'], exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6352f83f",
   "metadata": {},
   "source": [
    "`os.path.join`将创建一个路径，包含目录名和货架文件名`captions`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f5aeb7ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'photo_info/captions'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_file = os.path.join(config['data_dir'], 'captions')\n",
    "db_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbb08679",
   "metadata": {},
   "source": [
    "现在我们可以使用`shelve.open`方法打开该货架文件。实参`c`表示文件如果不存在，则自动创建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "291ea875",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<shelve.DbfilenameShelf at 0x7f4fcd248990>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import shelve\n",
    "\n",
    "db = shelve.open(db_file, 'c')\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e4a2fb3",
   "metadata": {},
   "source": [
    "返回值是一个`DbfilenameShelf`对象，俗称shelf货架对象。\n",
    "\n",
    "shelf对象在许多方面表现地与字典类似。例如我们可以使用方括号运算符添加一项，从键映射到值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b3f6d6ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = 'jan-2023/photo1.jpg' \n",
    "db[key] = 'Cat nose'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36fd5e3a",
   "metadata": {},
   "source": [
    "在这个例子里，键是图像文件的路径，值是一个描述图像内容的字符串。\n",
    "\n",
    "我们也使用方括号运算符查找键，获得对应的值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "d4e06b19",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Cat nose'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "value = db[key]\n",
    "value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9b252a7",
   "metadata": {},
   "source": [
    "如果你对现有的键进行再次赋值，`shelve`将替换旧的值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "89a0936c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Close up view of a cat nose'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db[key] = 'Close up view of a cat nose'\n",
    "db[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003eacbc",
   "metadata": {},
   "source": [
    "有些字典方法（如`keys`, `values`和`items`）对货架对象也有效。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7d0c4cff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['jan-2023/photo1.jpg']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a9db327d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Close up view of a cat nose']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056e0bd9",
   "metadata": {},
   "source": [
    "我们可以使用`in`运算符检查一个键是否在货架中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cc81ed95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key in db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "982740b4",
   "metadata": {},
   "source": [
    "我们可以使用`for`循环遍历所有的键。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "68ff774e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jan-2023/photo1.jpg : Close up view of a cat nose\n"
     ]
    }
   ],
   "source": [
    "for key in db:\n",
    "    print(key, ':', db[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67a0ddc",
   "metadata": {},
   "source": [
    "和其他文件一样，你可以在完成操作后关闭数据库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "2b411885",
   "metadata": {},
   "outputs": [],
   "source": [
    "db.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1e08b41",
   "metadata": {},
   "source": [
    "现在如果我们列出数据目录下的内容，我们将看到2个文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "4806720c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 当你打开shelve文件时，会创建一个后缀`.bak`的备份文件。\n",
    "# 如果你运行多次笔记，你可能会看到这个文件。\n",
    "# 这个单元格将移除它，以便笔记中的输出正确。\n",
    "\n",
    "!rm -f photo_info/captions.bak"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "59eb2dde",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['captions.dat', 'captions.dir']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(config['data_dir'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97453006",
   "metadata": {},
   "source": [
    "`captions.dat`包含我们刚才存储的数据。`captions.dir`包含组织数据库的信息，让数据读写更高效。\n",
    "\n",
    "后缀`dir`代表\"目录directory\"，但不会影响我们处理的目录。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0033e4f5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 存储数据结构\n",
    "\n",
    "在先前的例子中，货架的键和值都是字符串，但我们可以存储列表、字典等数据结构。\n",
    "\n",
    "让我们回忆[11章](chap11.ipynb)中的变位词练习。我们创建了一个字典，存储一组字母对应的变位词列表。例如，键`'opst'`映射到列表`['opts', 'post', 'pots', 'spot', 'stop', 'tops']`。\n",
    "\n",
    "我们将使用以下函数将这些字母存储为一个字符串。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "8819b211",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sort_word(word):\n",
    "    return ''.join(sorted(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c24316c",
   "metadata": {},
   "source": [
    "以下是一个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "fc8f1434",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'opst'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word = 'pots'\n",
    "key = sort_word(word)\n",
    "key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5d7555",
   "metadata": {},
   "source": [
    "现在让我们打开一个叫作`anagram_map`的货架。实参`'n'`表示我们总是创建一个新的空货架，即使该货架已经存在。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f2f17f05",
   "metadata": {},
   "outputs": [],
   "source": [
    "db = shelve.open('anagram_map', 'n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b495e79",
   "metadata": {},
   "source": [
    "现在我们可以像这样添加一项："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "b918dcfe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pots']"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db[key] = [word]\n",
    "db[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f058dff1",
   "metadata": {},
   "source": [
    "在这个项中，键是一个字符串，值是一个字符串的列表。\n",
    "\n",
    "现在假设我们找到另一个变位词（比如`tops`）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "e0d728e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'opst'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word = 'tops'\n",
    "key = sort_word(word)\n",
    "key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a215367d",
   "metadata": {},
   "source": [
    "这个键与之前的相同，所以我们想要添加第2个单词到相同的字符列表中。如果`db`是一个字典，我们会这样追加："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "12e39086",
   "metadata": {},
   "outputs": [],
   "source": [
    "db[key].append(word)          # 错误"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1a83f39",
   "metadata": {},
   "source": [
    "但如果我们执行该语句，检查货架中的键，看起来它并没有被更新。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "b0dc42e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pots']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a56de9",
   "metadata": {},
   "source": [
    "问题在于：当我们查询键是，我们获得一个字符串列表，但如果我们修改了列表，它不会影响货架。\n",
    "\n",
    "如果我们想要更新货架，我们必须读取旧值，更新，然后重新赋值给货架。\n",
    "\n",
    "译注：也就是说，shelf[key]返回的实际是这个值的副本，尝试对其进行追加、弹出等操作是没有意义或错误的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "b4105610",
   "metadata": {},
   "outputs": [],
   "source": [
    "anagram_list = db[key]\n",
    "anagram_list.append(word)\n",
    "db[key] = anagram_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a054f537",
   "metadata": {},
   "source": [
    "现在货架的值被更新了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "4f737cd9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pots', 'tops']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca02c0d7",
   "metadata": {},
   "source": [
    "作为练习，你可以完成这个示例，将所有变位词保存到货架中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "5395780f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "db.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1336e56",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 检查相等的文件\n",
    "\n",
    "现在让我们回到本章的目标：搜索有相同数据的不同文件。一种方式是读取每个文件的内容并进行比较。\n",
    "\n",
    "对于图像文件，我们必须以`'rb'`模式打开，其中`'r'`表示我们以只读的方式打开，而`'b'`则表示**二进制模式 binary mode**。二进制模式下内容不被解释为文本，而是处理为字节序列。\n",
    "\n",
    "以下是打开和读取图像文件的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a3104fc7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytes"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path1 = 'photos/jan-2023/photo1.jpg'\n",
    "data1 = open(path1, 'rb').read()\n",
    "type(data1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36687da2",
   "metadata": {},
   "source": [
    "`read`方法的结果是一个字节串`bytes`对象，包含一系列字节。\n",
    "\n",
    "总体而言图像文件的内容不是人类可读的，但如果我们读取第2个文件，我们可以使用`==`运算符进行比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "fd6dd21d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path2 = 'photos/jan-2023/photo2.jpg'\n",
    "data2 = open(path2, 'rb').read()\n",
    "data1 == data2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36006cd4",
   "metadata": {},
   "source": [
    "这两个文件不相等。\n",
    "\n",
    "让我们把目前使用的代码进行封装。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b1c8643c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def same_contents(path1, path2):\n",
    "    data1 = open(path1, 'rb').read()\n",
    "    data2 = open(path2, 'rb').read()\n",
    "    return data1 == data2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e08850",
   "metadata": {},
   "source": [
    "如果我们只有2个文件，这个函数是个好主意。但如果我们有大量的文件，我们想要知道它们是否包含相同数据，比较每对文件是低效的。\n",
    "\n",
    "备选方案是使用**哈希函数hash function**，接受文件的内容，计算文件的**摘要digest**，这个摘要通常是一个大整数。如果两个文件包含相同的数据，则必然有相同的摘要。如果两个文件不相同，则摘要*几乎*总是不同。\n",
    "\n",
    "`hashlib`模块提供了几个哈希函数，我们将使用其中的`md5`函数。首先使用`hashlib.md5`来创建一个`HASH`对象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "c03198fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_hashlib.HASH"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import hashlib\n",
    "\n",
    "md5_hash = hashlib.md5()\n",
    "type(md5_hash)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8e480f0",
   "metadata": {},
   "source": [
    "`HASH`对象提供`update`方法，接受文件的内容作为实参。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "1567ec9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "md5_hash.update(data1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86bd6bc0",
   "metadata": {},
   "source": [
    "现在我们可以使用`hexdigest`方法获得一个十六进制数字的字符串。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "b2e61e76",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'aa1d2fc25b7ae247b2931f5a0882fa37'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "digest = md5_hash.hexdigest()\n",
    "digest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9b4b347",
   "metadata": {},
   "source": [
    "以下函数封装了这些步骤："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "330def4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def md5_digest(filename):\n",
    "    data = open(filename, 'rb').read()\n",
    "    md5_hash = hashlib.md5()\n",
    "    md5_hash.update(data)\n",
    "    digest = md5_hash.hexdigest()\n",
    "    return digest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "914b85b2",
   "metadata": {},
   "source": [
    "如果我们对另一个文件内容进行哈希，我们可以确认获得了不同的摘要。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "eae2d207",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'6a501b11b01f89af9c3f6591d7f02c49'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filename2 = 'photos/feb-2023/photo2.jpg'\n",
    "md5_digest(filename2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deeabdde",
   "metadata": {},
   "source": [
    "现在我们已经准备好寻找相同的文件了。最后一步是搜索目录并找到所有的图像文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "129475df",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 漫游目录\n",
    "\n",
    "以下函数接受一个目录为实参，作为要搜索的路径。这个函数使用`listdir`遍历目录中的内容。如果找到文件，将完整路径打印;如果找到目录，递归调用自身，搜索子目录。\n",
    "\n",
    "译注：`os.walk`函数执行类似的功能，但输出的形式更适合迭代步骤。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "c84a64db",
   "metadata": {},
   "outputs": [],
   "source": [
    "def walk(dirname):\n",
    "    for name in os.listdir(dirname):\n",
    "        path = os.path.join(dirname, name)\n",
    "\n",
    "        if os.path.isfile(path):\n",
    "            print(path)\n",
    "        elif os.path.isdir(path):\n",
    "            walk(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faaf5b25",
   "metadata": {},
   "source": [
    "我们可以像这样使用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "f1c60f6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "photos/notes.txt\n",
      "photos/feb-2023/photo1.jpg\n",
      "photos/feb-2023/photo2.jpg\n",
      "photos/mar-2023/photo1.jpg\n",
      "photos/mar-2023/photo2.jpg\n",
      "photos/jan-2023/photo3.jpg\n",
      "photos/jan-2023/photo1.jpg\n",
      "photos/jan-2023/photo2.jpg\n"
     ]
    }
   ],
   "source": [
    "walk('photos')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "897c66bf",
   "metadata": {},
   "source": [
    "结果的输出顺序由操作系统的实现细节决定。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7853f18",
   "metadata": {},
   "source": [
    "## 调试\n",
    "\n",
    "当你读写文件时，你可能会遇到与空白符有关的问题。这些问题可能很难调试，因为空白符通常不可见。\n",
    "\n",
    "例如，以下是一个包含空格，制表符`\\t`，换行符`\\n`的字符串。当打印字符串时，我们无法看到空白符。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "15d7425a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 2\t 3\n",
      " 4\n"
     ]
    }
   ],
   "source": [
    "s = '1 2\\t 3\\n 4'\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49bbebe6",
   "metadata": {},
   "source": [
    "内建的函数`repr`能够帮你接受一个对象，返回其字符串表示形式。对于字符串，它将用反斜杠来表示空白符。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "61feff85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'1 2\\t 3\\n 4'\n"
     ]
    }
   ],
   "source": [
    "print(repr(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "790cf8dd",
   "metadata": {},
   "source": [
    "这对于调试很有帮助。\n",
    "\n",
    "你可能遇到的另一个问题在于，不同的操作系统使用不同的字符标识换行符。有些系统使用`\\n`，有些使用`\\r`，有些混合使用。如果你在不同的系统上移动文件，这种不一致可能会导致问题。\n",
    "\n",
    "文件大小写也可能是问题。在macOS和UNIX系统中，文件名可以包含大写和小写字母，数字和多数符号；而许多Windows应用会忽略大小写，且不允许部分符号。\n",
    "\n",
    "译注：在macOS和UNIX系统中的文件名是大小写敏感的，而Windows的NTFS文件系统默认不敏感。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf063639",
   "metadata": {},
   "source": [
    "## 术语表\n",
    "- **临时的ephemeral**：临时程序通常运行时间短语，在运行结束后数据会丢失；\n",
    "- **持久的persistent**：持久程序可能会永远运行，会存储至少部分数据；\n",
    "- **目录directory**：文件和其他目录的集合；\n",
    "- **当前工作目录current working directory**：程序默认使用的目录，除非明确指定其他目录；\n",
    "- **路径path**：标识一系列目录（通常跟随一个文件）的字符串；\n",
    "- **相对路径 relative path**：从当前工作目录或指定目录开始的路径；\n",
    "- **绝对路径 absolute path**：不依赖于当前目录的路径；\n",
    "- **f字符串 f-string**：字母`f`在引号之前的字符串，花括号内包含若干个表达式；\n",
    "- **配置数据configuration data**：通常存储于文件中的数据，指定程序如何执行；\n",
    "- **序列化serialization**：将对象转化为字符串；\n",
    "- **反序列化deserialization**：将字符串转化为对象；\n",
    "- **数据库database**：有效组织内容以高效执行特定操作的文件；\n",
    "- **键值存储 key-value stores**：以键-值对形式组织内容的数据库；\n",
    "- **二进制模式 binary mode**：一种打开文件的方式，将内容解释为字节串，而不是字符串；\n",
    "- **哈希函数 hash function**：针对一个对象，计算一个整数，这个整数有时被称为摘要；\n",
    "- **摘要digest**：哈希函数的结果，常用于检查两个对象是否相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67941fdd",
   "metadata": {},
   "source": [
    "## 练习"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd885ba1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 这个单元格让Jupyter在出现运行时故障时提供更多调试信息。\n",
    "# 在进行练习前先运行本单元格。\n",
    "\n",
    "%xmode Verbose"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a537173",
   "metadata": {},
   "source": [
    "### 询问虚拟助手\n",
    "\n",
    "本章有几个话题并没有详细解释。以下是几个可以询问的问题：\n",
    "\n",
    "- 临时程序与持久程序的区别是什么？\n",
    "- 有哪些程序是持久程序？\n",
    "- 相对路径和绝对路径的区别是什么？\n",
    "- `yaml`模块中为什么有`load`和`safe_load`两种函数？\n",
    "- 当使用Python中的shelf对象时，以`dat`和`dir`为后缀的文件名分别是什么？\n",
    "- 什么是哈希函数？\n",
    "- 什么是MD5摘要？\n",
    "- 和以前一样，如果你被以下练习卡住，考虑询问虚拟助手。除了问题，你还可以复制本章相应的代码。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7586e1e9",
   "metadata": {},
   "source": [
    "### 练习\n",
    "\n",
    "编写函数`replace_all`，要求接受一个模式字符串，一个替换字符串和两个文件名。\n",
    "\n",
    "它应该读取第1个文件，写入第2个文件（当需要时创建第2个文件）。如果模式字符串在内容中，应该被替换字符串替换。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85844afb",
   "metadata": {},
   "source": [
    "以下是函数的框架："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e598e70",
   "metadata": {},
   "outputs": [],
   "source": [
    "def replace_all(old, new, source_path, dest_path):\n",
    "    # 读取源文件的内容\n",
    "    reader = open(source_path)\n",
    "\n",
    "    # 替换旧字符串\n",
    "    \n",
    "    # 将结果写入到目标文件中\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3774d1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这作答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f37245c",
   "metadata": {},
   "source": [
    "要测试函数，读取文件`photos/notes.txt`，将`'photos'`替换为`'images'`，并将结果写入到文件`photos/new_notes.txt`中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b80dfd8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "source_path = 'photos/notes.txt'\n",
    "open(source_path).read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b67579f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dest_path = 'photos/new_notes.txt'\n",
    "old = 'photos'\n",
    "new = 'images'\n",
    "replace_all(old, new, source_path, dest_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d990225",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "open(dest_path).read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2589a4",
   "metadata": {},
   "source": [
    "### 练习\n",
    "\n",
    "在上文中，我们使用`shelve`模块创建一个键-值存储，将排序后的字母与对应的变位词映射。编写函数`add_word`，接受一个字符串和一个货架对象，存储某个文本内容中的所有变位词。\n",
    "\n",
    "首先，它应该对单词中的字母排序，作为一个键。如果键不在货架中，则创建一个列表作为值，将原始单词作为列表的元素，更新货架；否则在现有的列表上追加单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c2fff95",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这作答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07cb454f",
   "metadata": {
    "tags": []
   },
   "source": [
    "你可以使用这个循环测试你的函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8008cde6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "download('https://gitee.com/regentsai/Think_Python_3e_CN/blob/master/words.txt');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6b5e51a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "word_list = open('words.txt').read().split()\n",
    "\n",
    "db = shelve.open('anagram_map', 'n')\n",
    "for word in word_list:\n",
    "    add_word(word, db)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40342366",
   "metadata": {
    "tags": []
   },
   "source": [
    "如果一切顺利，你应该能够找到键`'opst'`，以及对应的单词："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffefe13e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "db['opst']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7eb54fbc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for key, value in db.items():\n",
    "    if len(value) > 8:\n",
    "        print(value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac784df7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "db.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228e977c",
   "metadata": {},
   "source": [
    "### 练习\n",
    "\n",
    "在大量文件中，可能有重复的文件存储于不同的目录或者有不同的文件名。这个练习意在找出重复文件。\n",
    "\n",
    "例如，我们将处理`photos`目录中的图像文件。\n",
    "\n",
    "以下是处理方式：\n",
    "\n",
    "- 使用`walk`函数，搜索这个目录中指定后缀名的文件，后缀名由`config['extensions']`规定；\n",
    "- 对于每个文件，使用`md5_digest`函数计算内容的摘要；\n",
    "- 使用货架，保存每个摘要与对应路径列表的映射；\n",
    "- 搜索货架中映射到多个路径的摘要；\n",
    "- 如果找到这样的文件，使用`same_contents`函数确认文件包含相同的数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f5365da",
   "metadata": {},
   "source": [
    "1. 要识别图像文件，编写函数`is_image`，接受一个路径和文件拓展名列表，当文件拓展名在指定拓展名中时返回`True`。提示：使用`os.path.splitext`函数，或者询问虚拟助手编写这个程序。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03b6acf9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这作答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde4b5c8",
   "metadata": {
    "tags": []
   },
   "source": [
    "你可以使用`doctest`测试你的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21c33092",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from doctest import run_docstring_examples\n",
    "\n",
    "def run_doctests(func):\n",
    "    run_docstring_examples(func, globals(), name=func.__name__)\n",
    "\n",
    "run_doctests(is_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a7c8f49",
   "metadata": {},
   "source": [
    "2. 编写函数`add_path`，接受一个路径和货架，使用`md5_digest`函数计算文件的摘要，然后更新货架。如果摘要已经在货架上，追加文件名至列表中；否则新建一个文件名列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31c056a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这作答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08223a21",
   "metadata": {},
   "source": [
    "3. 基于`walk`编写函数`walk_images`，接受一个目录，遍历目录中的文件和子目录。对于每个文件使用`is_image`检查是否是一个图像文件，使用`add_path`添加到货架。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e54d6993",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这作答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ea76a52",
   "metadata": {},
   "source": [
    "一切准备就绪后，使用以下程序创建货架，搜索`photos`目录，在货架中添加路径，检查同一个摘要是否对应多个文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c31ba5b6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "db = shelve.open('photos/digests', 'n')\n",
    "walk_images('photos')\n",
    "\n",
    "for digest, paths in db.items():\n",
    "    if len(paths) > 1:\n",
    "        print(paths)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "102d4d10",
   "metadata": {},
   "source": [
    "你应该找到2个文件有相同的摘要，使用`same_contents`检查是否包含相同的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7c7e679",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这作答"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6a6b5a7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "09ff2e33",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm anagram* camel-spotting-book.txt config.yaml\n",
    "!rm -R photo_info # 删除本章创建的文件"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f4edf8",
   "metadata": {
    "tags": []
   },
   "source": [
    "[Think Python: 3rd Edition](https://allendowney.github.io/ThinkPython/index.html)\n",
    "\n",
    "Copyright 2024 [Allen B. Downey](https://allendowney.com)\n",
    "\n",
    "Code license: [MIT License](https://mit-license.org/)\n",
    "\n",
    "Text license: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
